A Surprise Triggered Adaptive and Reactive (STAR) Framework for Online Adaptation in Non-stationary Environments
نویسندگان
چکیده
We consider the task of developing an adaptive autonomous agent that can interact with non-stationary environments. Traditional learning approaches such as Reinforcement Learning assume stationary characteristics over the course of the problem, and are therefore unable to learn the dynamically changing settings correctly. We introduce a novel adaptive framework that can detect dynamic changes due to non-stationary elements. The Surprise Triggered Adaptive and Reactive (STAR) framework is inspired by human adaptability in dealing with daily life changes. An agent adopting the STAR framework consists primarily of two components, Adapter and Reactor. The Reactor chooses suitable actions based on predictions made by a model of the environment. The Adapter observes the amount of “surprisingness” and triggers the generation of new models accordingly. Preliminary experimental results show that STAR agents are competitive in performance as compared with current approaches, while being much more costeffective by avoiding the negative effects of historical data. Furthermore, since response and adaptability are decoupled in the framework, the adaptive component can benefit other autonomous agents in a variety of domains with nonstationary environments.
منابع مشابه
The Time Adaptive Self Organizing Map for Distribution Estimation
The feature map represented by the set of weight vectors of the basic SOM (Self-Organizing Map) provides a good approximation to the input space from which the sample vectors come. But the timedecreasing learning rate and neighborhood function of the basic SOM algorithm reduce its capability to adapt weights for a varied environment. In dealing with non-stationary input distributions and changi...
متن کاملA new adaptive exponential smoothing method for non-stationary time series with level shifts
Simple exponential smoothing (SES) methods are the most commonly used methods in forecasting and time series analysis. However, they are generally insensitive to non-stationary structural events such as level shifts, ramp shifts, and spikes or impulses. Similar to that of outliers in stationary time series, these non-stationary events will lead to increased level of errors in the forecasting pr...
متن کاملAdaptive Variational Particle Filtering in Non-stationary Environments
Online convex optimization is a sequential prediction framework with the goal to track and adapt to the environment through evaluating proper convex loss functions. We study efficient particle filtering methods from the perspective of such framework. We formulate an efficient particle filtering methods for non-stationary environment by making connections with the online mirror descent algorithm...
متن کاملAdaptation in Constant Utility Non-Stationary Environments
Environments that vary over time present a fundamental problem to adaptive systems. Although in the worst case there is no hope of eeective adaptation, some forms environmental variability do provide adaptive opportunities. We consider a broad class of non-stationary environments, those which combine a variable result function with an invariant utility function, and demonstrate via simulation t...
متن کاملDesign of a Model Reference Adaptive Controller Using Modified MIT Rule for a Second Order System
Sometimes conventional feedback controllers may not perform well online because of the variation in process dynamics due to nonlinear actuators, changes in environmental conditions and variation in the character of the disturbances. To overcome the above problem, this paper deals with the designing of a controller for a second order system with Model Reference Adaptive Control (MRAC) scheme usi...
متن کامل